Robot system and control method of robot system for taking out workpieces loaded in bulk

ABSTRACT

A controller includes a generation unit that generates three-dimensional information, a deletion unit that deletes information about at least part of the measurement points in the three-dimensional information, and a determination unit that determines a state where the robot grips a workpiece. The generation unit generates first three-dimensional information before the robot implements an operation of gripping a target workpiece. The generation unit generates second three-dimensional information after the robot has implemented an operation of lifting the target workpiece. The deletion unit generates third three-dimensional information in which information about measurement points in the second three-dimensional information is deleted from the first three-dimensional information. The determination unit determines whether or not the workpiece in the third three-dimensional information matches the target workpiece.

RELATED APPLICATIONS

The present application claims priority of Japanese Application Number2018-096446, filed May 18, 2018, the disclosure of which is herebyincorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a robot system and a control method ofa robot system for taking out workpieces loaded in bulk.

2. Description of the Related Art

When many workpieces are contained in a case such as a container, theworkpieces may be arranged in a manner in which intervals between theworkpieces and orientations of the workpieces are irregular. In otherwords, the workpieces may be loaded in bulk inside the case. In thetechnique of related art, a robot system that includes a robot takingout the workpieces from this case is known (for example, JapaneseUnexamined Patent Publication No. 2010-69542A).

The workpieces loaded in bulk are contained at various intervals andwith various orientations. Accordingly, the workpieces are preferablytaken out after a position and an orientation of the workpiece aredetected. In the technique of the related art, it is known that athree-dimensional sensor is used for detecting the positions and theorientations of the workpieces (for example, Japanese Unexamined PatentPublication No. 2010-120141A and Japanese Unexamined Patent PublicationNo. 2017-64817A). In this method, the three-dimensional sensor capturesan image of workpieces arranged in the case. The image captured by thethree-dimensional sensor can be used for generating information such asa group of three-dimensional points of the workpieces loaded in bulk.Furthermore, the position and the orientation of the workpiece isdetected, and the position and the orientation of the robot can becontrolled based on the position and the orientation of the workpiece.

A hand having a plurality of craws for clamping the workpiece can beused as an end effector that grips the workpiece. Unfortunately, thehand having the plurality of craws can only grip certain parts of theworkpiece, and thus may not be capable of efficiently taking out theworkpieces loaded in bulk. For example, a hand that grips a front sideof a workpiece may fail to grip a back surface part of the workpiece ina case where the workpiece is flipped upside down. As a result, anoperation to take out all the workpieces may fail to be fullyaccomplished.

Preferably, the end effector is capable of gripping the workpiece invarious directions. In other words, preferably, a hand that can grip theworkpiece without being setting orientation of the hand relative to theworkpiece is used. In the technique of the related art, it is known thata magnetic hand that has a hemispherical tip portion and attracts aworkpiece with an electromagnet is used (for example, JapaneseUnexamined Patent Publication No. 2015-171749A).

SUMMARY OF THE INVENTION

In a state where workpieces are loaded in bulk in the case, a hand mayfail to grip a workpiece when the robot implements an operation oftaking out the workpiece from the case. Alternatively, the robot maytake out a workpiece different from a desired workpiece. This results ina longer cycle time. For example, when a robot implements an operationof conveying a workpiece to a next step without gripping the workpiece,a time is wasted for implementing an operation for conveying workpieceand an operation for returning to the original position. Thus,preferably, it is checked whether or not the desire workpiece is grippedbefore the workpiece is gripped by the hand and conveyed to the nextstep. In other words, a state where the workpiece is gripped ispreferably checked.

A device that detects a change in mass may be provided between the wristand the hand of the robot in order to check the state where theworkpiece is gripped (for example, Japanese Unexamined PatentPublication No. 2015-171749A described above). The state where theworkpiece is gripped can be checked by measuring the difference betweenthe mass of the hand before the workpiece is gripped and the mass of thehand after the workpiece is gripped. Alternatively, a photoelectricsensor may be attached to the hand or the like. When the workpiece isgripped, light output from the photoelectric sensor is reflected by asurface of the workpiece. When the photoelectric sensor receives thereflected light, the workpiece can be determined to be gripped. However,these methods are disadvantageous in that an additional device forchecking the state where the workpiece is gripped is required.Furthermore, the device that detects a change in mass may fail to detecta workpiece with a small mass, in a case where a movement part of therobot involves large friction.

In a case where the hand that can grip a workpiece without setting theorientation of the hand relative to the workpiece is used, the workpieceis gripped by the hand while being in various orientations. This leadsto a disadvantage that it is difficult to detect the workpiece with thephotoelectric sensor.

For example, a magnetic hand can be employed as the hand and thephotoelectric sensor can be employed as a device for detecting theworkpiece. A workpiece may be in any orientation relative to the handwhen the workpiece is gripped. Thus, the light emitted from thephotoelectric sensor may not be reflected depending on the orientationof the workpiece gripped by the hand. In such a case, the hand isdetermined to be not gripping the workpiece despite the fact that theworkpiece is actually being gripped by the hand. Furthermore, when thephotoelectric sensor is used, there is a disadvantage that whether asingle workpiece is gripped or two or more workpieces are gripped cannotbe determined.

A first aspect of the present disclosure is a robot system that takesout each of workpieces loaded in bulk inside a case. The robot systemincludes a hand that grips workpieces, a robot that moves the hand, athree-dimensional sensor that acquires information about distances tothe workpieces, and a controller that controls the hand and the robot.The controller includes an operation control unit that sends operationcommands to the hand and the robot, and a generation unit that generatesthree-dimensional information including information about positions ofmeasurement points set to the workpieces based on an output from thethree-dimensional sensor. The controller includes a storage unit thatstores three-dimensional shape data of the workpieces and a detectionunit that detects positions and orientations of the workpieces byimplementing model matching in which the three-dimensional informationabout the workpieces is compared with the three-dimensional shape data.The controller includes a selection unit that selects a target workpieceto be taken out by the robot based on the positions and the orientationsof the workpieces. The controller includes a deletion unit that deletesinformation about at least part of the measurement points included inthe three-dimensional information, and a determination unit thatdetermines a state after the robot implements an operation for liftingthe workpiece. The generation unit generates first three-dimensionalinformation based on an output from the three-dimensional sensor thathas captured an image of the workpieces before the robot implements anoperation of gripping the target workpiece. The operation control unitcontrols the robot so as to stop after implementing an operation ofgripping and lifting the target workpiece selected based on the firstthree-dimensional information. The generation unit generates secondthree-dimensional information based on an output from thethree-dimensional sensor that has captured an image of the workpiecesafter the robot implements the operation of lifting the targetworkpiece. The deletion unit compares a position of the measurementpoint in the first three-dimensional information with a position of themeasurement point in the second three-dimensional information. Thedeletion unit detects, as a specified measurement point, the measurementpoint in the first three-dimensional information from which themeasurement point in the second three-dimensional information is presentwithin a predetermined distance range. The deletion unit generates thirdthree-dimensional information by deleting information about thespecified measurement point from the first three-dimensionalinformation. The detection unit detects a position and an orientation ofa workpiece included in the third three-dimensional information. Thedetermination unit determines whether or not the workpiece included inthe third three-dimensional information matches the target workpiece.When the workpiece included in the third three-dimensional informationmatches the target workpiece, the operation control unit controls therobot so as to convey the workpiece gripped by the hand to apredetermined conveyance destination position.

A second aspect of the present disclosure is a control method of a robotsystem that includes a robot and a hand, and takes out each ofworkpieces loaded in bulk in a case. The control method includes anoperation control step of changing a position and an orientation of therobot. The control method includes an image capturing step of capturingan image of the workpieces with a three-dimensional sensor that acquiresinformation about distances to the workpieces, and a generation step ofgenerating three-dimensional information including information aboutpositions of measurement points set to the workpieces based on an outputfrom the three-dimensional sensor. The control method includes adetection step of detecting the positions and the orientations of theworkpieces by implementing model matching in which the three-dimensionalinformation about the workpieces is compared with three-dimensionalshape data of the workpieces. The control method includes a selectionstep of selecting a target workpiece to be taken out by the robot basedon the positions and the orientations of the workpieces. The controlmethod includes a deletion step of deleting information about at leastpart of the measurement points included in the three-dimensionalinformation, and a determination step of determining a state after therobot implements an operation of lifting the workpiece. The generationstep includes generating first three-dimensional information based on anoutput from the three-dimensional sensor that has captured an image ofthe workpiece before the robot implements an operation of gripping thetarget workpiece. The operation control step includes a step ofcontrolling the robot so as to stop after implementing an operation ofgripping and lifting the target workpiece selected based on the firstthree-dimensional information. The generation step includes generatingsecond three-dimensional information based on an output from thethree-dimensional sensor that has captured an image of the workpiecesafter the robot has implemented the operation of lifting the targetworkpiece. The deletion step includes a step of comparing a position ofthe measurement point in the first three-dimensional information with aposition of the measurement point in the second three-dimensionalinformation. The deletion step includes a step of detecting, as aspecified measurement point, the measurement point in the firstthree-dimensional information from which the measurement point in thesecond three-dimensional information is present within a predetermineddistance range. The deletion step includes a step of generating thirdthree-dimensional information by deleting information about thespecified measurement point from the first three-dimensionalinformation. The detection step includes a step of detecting a positionand an orientation of a workpiece included in the thirdthree-dimensional information. The determination step includes a step ofdetermining whether or not the workpiece included in the thirdthree-dimensional information matches the target workpiece. Theoperation control step includes a step of controlling, when theworkpiece included in the third three-dimensional information matchesthe target workpiece, the robot so as to convey the workpiece gripped bythe hand to a predetermined conveyance destination position.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of a robot system of an embodiment.

FIG. 2 is a block diagram of the robot system of the embodiment.

FIG. 3 is a plan view of a container containing workpieces therein.

FIG. 4 is a partial cross-sectional view of a range sensor and acontainer of the embodiment.

FIG. 5 is a partial cross-sectional view of the container and aworkpiece of the present embodiment.

FIG. 6 is a diagram illustrating measurement points set on surfaces ofworkpieces.

FIG. 7 illustrates a first distance image before a workpiece is takenout.

FIG. 8 illustrates a second distance image after the workpiece is takenout.

FIG. 9 is a diagram illustrating specified measurement points to bedeleted from the first distance image.

FIG. 10 illustrates a third distance image generated by deletinginformation about the specified measurement points from the firstdistance image.

FIG. 11 is a first side view of a hand, a workpiece, and a photoelectricsensor of a first comparative example.

FIG. 12 is a second side view of the hand, the workpiece, and thephotoelectric sensor of the first comparative example.

FIG. 13 is a first side view of a hand, a workpiece, and a photoelectricsensor of a second comparative example.

FIG. 14 is a second side view of the hand, the workpiece, and thephotoelectric sensor of the second comparative example.

FIG. 15 illustrates a third side view of a hand, a workpiece, and aphotoelectric sensor of the second comparative example.

FIG. 16 is a flowchart of a control in the embodiment.

DETAILED DESCRIPTION

A robot system and a control method of the robot system according toembodiments will be described while referencing FIGS. 1 to 16. The robotsystem according to the present embodiment implements an operation oftaking out workpieces piled up in a case, and conveying the workpiecesto a predetermined conveyance destination position.

FIG. 1 is a perspective view of a robot system of the presentembodiment. The robot system 5 includes a robot 1 and a hand 2. Therobot system 5 includes a controller 4 that controls the robot 1 and thehand 2. The robot 1 of the present embodiment is an articulated robothaving a plurality of joints. The robot 1 includes an upper arm 11 and alower arm 12. The lower arm 12 is supported by a swivel base 13. Theswivel base 13 is supported by a base 14. The robot 1 includes a wrist15 coupled to an end portion of the upper arm 11. These components ofthe robot 1 are each formed to rotate about a predetermined rotationaxis. Note that the robot is not limited to the mode, and any robotcapable of changing the position and the orientation of the hand can beemployed.

The hand 2 is an operation tool that grips and releases the workpiece W.The hand 2 includes an electromagnet 2 a that produces attraction forceby magnetic force and an attraction member 2 b that attracts theworkpieces W with the magnetic force of the electromagnet 2 a. Theattraction member 2 b is formed to have a rod shape. The attractionmember 2 b is formed to have a spherical tip portion. The hand 2 of thepresent embodiment can grip the workpiece W without setting a positionand an orientation of the hand 2 relative to the workpiece W.

The workpieces W are arranged inside a container 9 serving as a case.The case may be any member that can accommodate the workpieces W, suchas a box, a bag, a basket, or the like. In the container 9, theworkpieces W are loaded in bulk. The container 9 is placed on a table81. The robot system 5 of the present embodiment takes out theworkpieces W one by one from the container 9, and conveys the workpiecesW to the predetermined conveyance destination position.

The robot system 5 includes a range sensor 6 serving as athree-dimensional sensor for detecting the position and the orientationof the workpiece W. The range sensor 6 acquires information about adistance to the workpiece W. The range sensor 6 is arranged at aposition to be capable of capturing an image of the workpiece W. In thepresent embodiment, the range sensor 6 is arranged above the container9. The range sensor 6 is fixed to a support member 83. The range sensor6 is controlled by the controller 4.

The range sensor 6 of the present embodiment is a stereo cameraincluding two cameras 61 and 62. The cameras 61 and 62 are each atwo-dimensional camera that can capture a two-dimensional image. Thecameras 61 and 62 may be any cameras including image element such as aCharge-Coupled Device (CCD) sensor or a Complementary Metal-OxideSemiconductor (CMOS) sensor. The relative positions of the two cameras61 and 62 are determined in advance. The range sensor 6 of the presentembodiment includes a projector 63 that projects light in a pattern,such as a stripe pattern, toward the workpieces W.

A reference coordinate system 37 that is immovable in response to achange in the position and the orientation of the robot 1 is set to therobot system 5 of the present embodiment. In the example illustrated inFIG. 1, the reference coordinate system 37 has the origin arranged atthe base 14 of the robot 1. The reference coordinate system 37 is alsoreferred to as a world coordinate system. The reference coordinatesystem 37 has an X axis, a Y axis, and a Z axis set to be orthogonal toeach other as coordinate axes. For example, a coordinate value (xb, yb,zb) of the reference coordinate system 37 can be set. Furthermore, Waxis serving as coordinate axis around the X axis, P axis serving ascoordinate axis around the Y axis, and R axis serving as coordinate axisaround the Z axis may be set. In this case, a coordinate value (xb, yb,zb, wb, pb, rb) of the reference coordinate system 37 can be set.

FIG. 2 is a block diagram of a robot system of the present embodiment.Referring to FIG. 1 and FIG. 2, the robot 1 includes a robot drivedevice that changes the position and the orientation of the robot 1. Therobot drive device includes robot drive motors 22 that drive componentssuch as an arm and a wrist. By driving the robot drive motor 22, theorientation of each component is changed.

The robot system 5 includes a hand drive device that drives the hand 2.The workpiece W of the present embodiment is formed of a magneticmaterial such as iron. The workpiece W is attracted to the attractionmember 2 b when the electromagnet 2 a of the hand 2 is driven.

The controller 4 includes an arithmetic processing device (computer)having a Central Processing Unit (CPU), a Random Access Memory (RAM), aRead Only Memory (ROM), or the like connected to the CPU via a bus. Anoperation program 41 that is generated in advance for operating therobot 1 is input to the controller 4. The operation program 41 is storedin the storage part 42. Furthermore, the storage unit 42 storesinformation about control such as a determination value and adetermination range. The robot system 5 of the present embodimentconveys the workpiece W based on the operation program 41. The robot 1can automatically convey the workpiece W to a predetermined conveyancedestination position.

The operation control unit 43 sends an operation command for driving therobot 1 based on the operation program 41 to a robot drive unit 45. Therobot drive unit 45 includes an electric circuit that drives the robotdrive motor 22. The robot drive unit 45 supplies electricity based onthe operation command to the robot drive motor 22. The operation controlunit 43 sends an operation command for driving the hand 2 based on theoperation program 41 to a hand drive unit 44. The hand drive unit 44includes an electric circuit that drives the electromagnet 2 a. The handdrive unit 44 supplies electricity to the electromagnet 2 a based on theoperation command. The operation control unit 43 sends an operationcommand for driving the range sensor 6 to the range sensor 6 based onthe operation program 41.

The robot 1 includes a status detector for detecting the position andthe orientation of the robot 1. The status detector of the presentembodiment includes a position detector 18 attached to the robot drivemotor 22 corresponding to the drive axis of a component such as an arm.The orientation of the component along each drive axis can be acquiredfrom an output from the position detector 18. For example, the positiondetector 18 detects a rotational angle when the robot drive motor 22 isdriven. The position and the orientation of the hand 2 are detectedbased on an output from the position detector 18.

The controller 4 includes an operation setting unit 52 that sets theoperation of the robot 1 based on the operation program 41. Theoperation setting unit 52 of the present embodiment implements controlfor selecting the workpiece W to be taken out from the container 9 andgripping the workpiece W with the hand 2. The operation setting unit 52detects the status of the workpiece W gripped by the hand 2 and controlsthe operation of the robot 1.

FIG. 3 is a plan view of the container containing workpieces therein.FIG. 4 is a partial cross-sectional view illustrating relationshipbetween a range sensor and a workpiece of the present embodiment. InFIG. 4, the range sensor 6 and the container 9 are illustrated to bearranged with a short distance in between for the sake of description. Aplurality of the workpieces W are arranged in the container 9. Theworkpiece W of the present embodiment is formed to have a cylindricalshape. In the present embodiment, a state where the workpieces W are notarranged at an equal distance or in the same orientation in at least onedirection of the container 9 is referred to as a state where theworkpieces W are loaded in bulk. In the present embodiment, theplurality of the workpieces W are arranged in the container 9 so thataxial directions of the workpieces W are different from each other. Inother words, the workpieces W are loaded so that a plurality ofworkpieces W are orientated randomly.

The range sensor 6 has an image capturing range 64 in which an image canbe captured. The cameras 61 and 62 are preferably arranged so that thecontainer 9 is placed within the image capturing range 64. In otherwords, the cameras 61 and 62 are preferably arranged to be capable ofcapturing an image including all the workpieces W that can be viewedfrom above the container 9.

FIG. 5 is a partial cross-sectional view of a hand and a container in anoperation of taking out a workpiece. The robot 1 is driven so as tobring the attraction member 2 b of the hand 2 into contact with theworkpiece W. The electromagnet 2 a of the hand 2 is driven, whereby theworkpiece W is attracted to the attraction member 2 b. The robot 1 isthen driven so as to move the hand 2, whereby the workpiece W can belifted as indicated by an arrow 91. In the present embodiment, theworkpiece W is gripped by the hand 2 in a random orientation.

Referring to FIG. 2 and FIG. 4, the controller 4 includes a generationunit 58 that generates three-dimensional information about the workpieceW based on an output from the range sensor 6. The generation unit 58processes images captured by the cameras 61 and 62. The generation unit58 can employ stereoscopy so as to generate three-dimensionalinformation about the workpiece W. The three-dimensional informationincludes information about positions of measurement points set onsurfaces of the workpieces W.

FIG. 6 is a diagram illustrating measurement points set on surfaces ofworkpieces. The generation unit 58 may employ any method in order to seta plurality of measurement points MP on the surfaces of the workpieces Was viewed from the range sensor 6. For example, referring FIG. 4, thegeneration unit 58 can set a plane 65 orthogonal to an optical axis 63 aof the projector 63. The generation unit 58 can set lines in a latticeform on the plane 65. For example, the generation unit 58 can setstraight lines extending in the direction of the X axis and the straightlines extending in the direction of the Y axis that are orthogonal toeach other. The generation unit 58 sets a straight line 66 connecting anintersection between these plurality of straight lines and the lenscenter point of the projector 63 to each other. The generation unit 58can set the measurement point MP at a point of intersection between thesurface of the workpiece W and the direction of the straight line 66.

The generation unit 58 calculates the distance from the range sensor 6to the measurement point MP based on parallax between two imagescaptured by the two cameras 61 and 62. The generation unit 58 can detectthe position of the measurement point MP based on the distance to themeasurement point MP and the orientation of the straight line 66. Forexample, the generation unit 58 can calculate the coordinate value ofthe measurement point MP in the reference coordinate system 37 andgenerate three-dimensional information.

The three-dimensional information includes information about thepositions of the plurality of measurement points MP. Thethree-dimensional information of the present embodiment includes thecoordinate value of the measurement point MP in the reference coordinatesystem 37. The three-dimensional information may include informationabout the distance from the range sensor 6 to the measurement point MPand a direction from the range sensor 6 to the measurement point MP. Forexample, the three-dimensional information is a distance image or athree-dimensional map. The distance image is an image with a color ordensity that changes in accordance with the distance from the rangesensor 6. The three-dimensional map includes a coordinate value of ameasurement point in a predetermined coordinate system or informationabout the distance and the direction from the range sensor to themeasurement point. The generation unit 58 of the present embodiment isincluded in the controller 4 that controls the robot 1, but theembodiment is not limited to this. An arithmetic processing device thatprocesses the images captured by the cameras 61 and 62 may be providedseparately from the controller 4.

Referring to FIG. 2, the detection unit 53 implements model matching inwhich the three-dimensional information about the workpiece W iscompared with the three-dimensional shape data so as to detect theposition and the orientation of the workpiece W. In the presentembodiment, three-dimensional shape data 46 about the workpiece Wgenerated in advance is stored in the storage unit 42. For example,three-dimensional data of the shape generated by a Computer Aided Design(CAD) device can be used as the three-dimensional shape data 46. Thegeneration unit 58 may detect the orientation of each of the workpiecesW based on the three-dimensional shape data 46 generated by the CADdevice.

Alternatively, an operator may employ a distance image of a workpiececaptured in a predetermined direction as the three-dimensional shapedata 46. The operator may employ distance images of a workpiece capturedin a plurality of directions for showing largely different shapes of theworkpiece, as the three-dimensional shape data 46. For example, theworkpiece W of the present embodiment has a cylindrical shape. Thus, thedistance image captured in a direction orthogonal to the axis of theworkpiece and the distance image of the workpiece W as viewed in theaxial direction may be employed as the three-dimensional shape data 46.Even when the direction of viewing the workpiece W is slightly shifted,the detection unit 53 can use such pieces of three-dimensional shapedata 46 so as to detect the position and the orientation of theworkpiece W.

When a plurality of types of workpieces are arranged in the case, theoperator generates the three-dimensional shape data 46 for each type ofworkpiece. The storage unit 42 can store a plurality of types ofthree-dimensional shape data 46. The operation setting unit 52 canimplement determination on each type of workpiece.

The operation setting unit 52 includes a selection unit 59 that selectsa target workpiece W to be picked up by the robot 1. The selection unit59 selects the target workpiece W based on the positions and theorientations of the workpieces W detected by the detection unit 53. Theselection unit 59 may implement any appropriate control in order toselect the target workpiece W. For example, the workpiece W closest tothe range sensor 6 may be set as the target workpiece W. Thus, theworkpiece W at the highest position may be selected one by one. Under acertain condition, the workpiece W at a low position may be selected.For example, the selection unit 59 may select the workpiece W in which aportion to be gripped is exposed largely so as to be easily gripped bythe hand 2.

The selection unit 59 may determine whether or not the workpiece W canbe gripped by the hand 2. For example, the attraction member 2 b of thehand 2 may be unable to be brought into contact with the workpiece Wwhen the workpiece W is close to a wall surface of the container 9. Insuch a case, the selection unit 59 may determine that the workpiece Wcannot be gripped.

The selection unit 59 calculates a target position and a targetorientation with which the robot 1 can grip the workpiece W, based onthe position and the orientation of the target workpiece W. Theselection unit 59 transmits the target position and the targetorientation of the robot 1 to the operation control unit 43. Theposition and the orientation of the robot 1 are changed to the targetposition and the target orientation. The workpiece W can be gripped withthe attraction member 2 b of the hand 2 excited after the robot 1reaches the target position and the target orientation. Then, theoperation control unit 43 changes the position and the orientation ofthe robot 1 so as to implement an operation of lifting the workpiece W.

The operation setting unit 52 includes a determination unit 55 thatdetermines a state after the robot 1 has lifted the workpiece W. Thedetermination unit 55 determines whether or not the workpiece W islifted in a desired state. When the robot 1 grips the workpiece W in adesired state, the determination unit 55 transmits an operation commandfor conveying the workpiece to the conveyance destination position tothe operation control unit 43. The operation control unit 43 controlsthe robot 1 so as to convey the workpiece to the conveyance destinationposition.

When the robot 1 does not grip the workpiece W in the desired state, thedetermination unit 55 transmits an operation command for returning theworkpiece W gripped by the hand 2 to an original position before beinggripped by the hand 2, to the operation control unit 43. Alternatively,when it is determined that the hand 2 does not grip the workpiece W, thedetermination unit 55 transmits an operation command for implementingcontrol from an operation for capturing an image of the workpiece.

Next, control for determining the state where the workpiece is grippedby the determination unit 55 and changing the operation of the robot 1in accordance with the state where the workpiece W is gripped isdescribed in detail.

FIG. 7 shows an example of first three-dimensional information generatedby a generation unit. FIG. 7 illustrates a first distance image 31captured by the range sensor 6 before the robot 1 is driven. In thisexample, a distance image that has a density varying in accordance withthe distance from the range sensor 6 is shown. In this distance image, apart closer to the range sensor 6 is shown with a lower density. Thefirst distance image 31 includes images 72 a, 72 b, and 72 c of aplurality of workpieces and an image 71 of a container.

In the present embodiment, the range sensor 6 captures an image of theworkpieces W before the robot 1 implements an operation of gripping thetarget workpiece W. The generation unit 58 generates the first distanceimage 31 based on an output from the range sensor 6. The first distanceimage 31 corresponds to the first three-dimensional informationgenerated based on an output from the range sensor 6 that has capturedan image of the workpiece W. The three-dimensional information of thepresent embodiment includes information about coordinate values of themeasurement points MP.

The detection unit 53 detects the position and the orientation of eachworkpiece W based on the first distance image 31. The selection unit 59selects the workpiece W to be taken out by the robot 1 based on thepositions and the orientations of the workpieces W. The operationcontrol unit 43 controls the robot 1 so as to grip the target workpieceW. The operation control unit 43 controls the robot 1 so as to stopafter implementing an operation of lifting the target workpiece W. Atthis time, the robot 1 takes out the target workpiece W from thecontainer 9 and moves to the outside of an image capturing range 64 ofthe range sensor 6. In other words, the position and the orientation ofthe robot 1 are changed so that the robot 1, the hand 2, and theworkpiece W gripped by the hand 2 are not included in the distance imageto be captured by the range sensor 6. The robot 1 arranges the workpieceW in the vicinity of the container 9, instead of conveying it to thetarget conveyance destination position.

In this example, the target workpiece W is a workpiece corresponding tothe image 72 a among the images 72 a, 72 b, and 72 c of the plurality ofworkpieces. Furthermore, in this example, the robot 1 lifts the targetworkpiece W only. Thus, the robot 1 does not lift any workpiece W otherthan the target workpiece.

Next, the range sensor 6 captures an image of the workpieces W arrangedin the container 9, after the robot 1 has implemented the operation oflifting the workpiece W. The generation unit 58 generates a seconddistance image as second three-dimensional information based on anoutput from the range sensor 6.

FIG. 8 shows an example of a distance image after the operation oflifting the target workpiece has been implemented. In the seconddistance image 32, the image 72 a is no longer included, since theworkpiece W corresponding to the image 72 a in FIG. 7 has been taken outfrom the container 9.

Referring to FIG. 2, the operation setting unit 52 of the presentembodiment includes a deletion unit 54 that deletes information about atleast part of the measurement points in the three-dimensionalinformation. The deletion unit 54 compares the position of themeasurement point MP in the first three-dimensional information with theposition of the measurement point MP in the second three-dimensionalinformation. The deletion unit 54 detects, as a specified measurementpoint, the measurement point in the first three-dimensional informationfrom which the measurement point in the second three-dimensionalinformation is present within a predetermined distance range. Thedeletion unit 54 generates third three-dimensional information in whichthe information about the specified measurement point is deleted fromthe first three-dimensional information.

FIG. 9 is a schematic view illustrating the measurement point in thefirst three-dimensional information generated before the workpiece isgripped and the measurement points in the second three-dimensionalinformation generated after the operation of lifting the workpiece hasbeen implemented. In the example illustrated in FIG. 9, the workpiece Wdifferent from the target workpiece is illustrated. Specifically, in thecontainer 9, a workpiece in which the position after the targetworkpiece is lifted is unchanged from the position before the targetworkpiece is lifted is illustrated.

If the workpieces remaining in the container 9 do not move when therobot 1 implements the operation of lifting the workpiece W, thepositions of measurement points MPa included in the firstthree-dimensional information match the positions of measurement pointMPb included in the second three-dimensional information. Even in such asituation, the positions of measurement points MPa may be slightlydifferent from the positions of measurement point MPb due to ameasurement error and the like. Furthermore, slight movements of theremaining workpieces in the container 9 as result of the operation oflifting the target workpiece by the robot 1 may not affect a next stepof taking out the next workpiece.

When the measurement point MPb included in the second three-dimensionalinformation is present in the vicinity of the measurement point MPaincluded in the first three-dimensional information, the deletion unit54 of the present embodiment sets the measurement point MPa to be thespecified measurement point. Then, the deletion unit 54 generates thethird three-dimensional information in which the information about thespecified measurement point is deleted from the first three-dimensionalinformation.

The determination range for deleting the measurement point MPa includedin the first three-dimensional information is determined in advance. Inthe present embodiment, the determination range in the X axis direction,the Y axis direction, and the Z axis direction in the referencecoordinate system is determined. In the example illustrated in FIG. 9, adetermination range 67 relating to the X axis direction and the Y axisdirection is illustrated. The measurement point MPb included in thesecond three-dimensional information is present in the determinationrange 67 with respect to the measurement point MPa included in the firstthree-dimensional information. Thus, the deletion unit 54 detects themeasurement point MPa as the specified measurement point. The deletionunit 54 deletes the specified measurement point from the firstthree-dimensional information. For example, the coordinate value of themeasurement point MPa included in the first distance image 31 isdeleted. In this manner, the deletion unit 54 generates the thirdthree-dimensional information in which the specified measurement pointis deleted from the first three-dimensional information.

In the example illustrated in FIG. 9, the measurement points MPbincluded in the second three-dimensional information are present in thevicinity of all of the measurement points MPa included in the firstthree-dimensional information. Thus, the deletion unit 54 designates allthe measurement points MPa to be the specified measurement points, anddeletes the measurement points MPa from the first distance image.

FIG. 10 shows an example of a third distance image generated by adeletion unit. Referring to FIG. 7, FIG. 8, and FIG. 10, in the thirddistance image 33 serving as the third three-dimensional information,the images 72 b and 72 c of the plurality of workpieces in which thepositions and the orientations are not changed are deleted from thefirst distance image 31. In the third distance image 33, the image 72 aof the workpiece taken out from the container 9 by the robot 1 remains.

The detection unit 53 implements model matching in which the thirddistance image 33 and the three-dimensional shape data 46 are comparedwith each other, whereby the position and the orientation of theworkpiece W corresponding to the image 72 a is detected.

Next, the determination unit 55 determines whether or not the workpieceW detected by the detection unit 53 match the target workpiece W. Thedetermination unit 55 acquires the position and the orientation of thetarget workpiece W from the selection unit 59. The determination unit 55determines that the workpiece included in the third distance image 33matches the target workpiece in a case where a difference between theposition and the orientation of the workpiece W included in the thirddistance image 33 and the position and the orientation of the targetworkpiece is within a predetermined determination range. When thedetection unit 53 detects a plurality of workpieces and a workpiecedifferent from the target workpiece is included, the determination unit55 determines that the workpiece included in the third three-dimensionalinformation does not match the target workpiece.

In the example illustrated in FIG. 10, the determination unit 55determines that the target workpiece W is taken out. Next, thedetermination unit 55 sends a command for conveying the workpiece Wgripped by the hand 2 to the predetermined conveyance destinationposition, to the operation control unit 43. The operation control unit43 controls the robot 1 so as to convey the workpiece W gripped by thehand 2 to the conveyance destination position.

In this manner, in the robot system 5 according to the presentembodiment, it can be determined whether or not the hand 2 grips thetarget workpiece through the comparison between the firstthree-dimensional information before the operation of lifting theworkpiece and the second three-dimensional information after theoperation of lifting the workpiece.

If the hand 2 grips a single workpiece W different from the targetworkpiece W when the robot implements the operation of lifting thetarget workpiece W, the information about the workpiece different fromthe target workpiece W remains in the third three-dimensionalinformation. The detection unit 53 detects the position and theorientation of the workpiece different from the target workpiece. Thedetermination unit 55 determines that the workpiece detected by thedetection unit 53 is different from the target workpiece. Furthermore,the hand 2 may grip the target workpiece together with a workpiece otherthan the target workpiece. In other words, the hand 2 may grip two ormore workpieces W. In such a case, the third three-dimensionalinformation includes information about a plurality of workpieces W. Thedetection unit 53 detects the positions and the orientations of theplurality of workpieces W. The determination unit 55 determines that theplurality of workpieces W detected by the detection unit 53 do not matchthe target workpiece.

When the hand 2 grips a workpiece different from the target workpiece orgrips two or more workpieces, the determination unit 55 sends a commandfor returning the workpiece(s) W gripped by the hand 2 to the vicinityof the original position(s), to the operation control unit 43. Theoperation control unit 43 controls to robot 1 so as to return theworkpiece(s) gripped by the hand 2 to the vicinity of the originalposition(s) before being gripped by the hand 2. In other words, theoperation control unit 43 implements the control for returning theworkpiece W to the vicinity of the position of the target workpiece inthe first three-dimensional information.

After the robot 1 returns the workpiece W to the vicinity of theoriginal position, the control for conveying the workpiece W isimplemented from the step of capturing the image of the workpieces W.Specifically, the range sensor 6 captures an image of the workpieces Warranged in the container 9, the detection unit 53 detects the positionsand the orientations of the workpieces W, and the selection unit 59selects the next target workpiece W to be taken out by the robot 1.Then, the controller 4 implements control similar to that describedabove in order to take out the next target workpiece W.

When the workpiece W is returned into the container 9, control may beimplemented so that the workpiece W gripped by the hand 2 is returned toa position different from the original position. However, this controlhas a possibility of causing load collapse in the container 9. BYadopting the control for returning the workpiece W gripped by the hand 2to the vicinity of the original position, a possibility of the loadcollapse as a result of returning the workpiece W can be reduced. Notethat the control for returning the workpiece W gripped by the hand 2 tothe original position may be implemented.

The detection unit 53 may fail to detect the position and theorientation of the workpiece W in the third three-dimensionalinformation. In other words, even when the model matching in which thethird three-dimensional information and the three-dimensional shape data46 are compared with each other is performed, the position and theorientation of the workpiece W in the third distance image 33illustrated in FIG. 10 may not be detected. For example, the hand 2 mayfail to grip the workpiece W and thus may not grip the workpiece W.Furthermore, when the operation of lifting the target workpiece W by therobot 1 is performed, the load collapse of the workpieces W do not occurin the container 9, and thus the positions and the orientations of allthe workpieces W may remain to be the same. Alternatively, the amount ofmovement of the position of the workpiece W may be extremely small, andthus the information about the measurement point may be deleted by thedeletion unit 54. In such a case, the third distance image 33 includesno image of the workpiece, and thus the detection unit 53 cannot detectany workpiece.

The determination unit 55 can detect the number of measurement pointsother than the specified measurement points, among the measurementpoints in the first three-dimensional information. The determinationunit 55 can determine that the hand 2 does not grip the workpiece W in acase where the number of the measurement points other than the specifiedmeasurement points is smaller than a predetermined determination valuerelated to the failure in the gripping. In other words, when almost allthe measurement points are deleted by the deletion unit 54, thedetermination unit 55 can determine that the positions and theorientations of all the workpieces W arranged in the container 9 do notchange. The determination unit 55 can determine that the workpieces W donot collapse as a result of the operation of taking out the workpiece W.A small value may be employed as the determination value related to thefailure to grip the workpiece. For example, a value smaller than thelower limit of the number of measurement points set to a singleworkpiece can be employed.

After the determination unit 55 determines that the hand 2 grips noworkpiece W, the controller 4 implements the control for conveying theworkpiece W from the operation for capturing the image of the workpieceW. In other words, the range sensor 6 captures an image of theworkpieces W arranged in the container 9, the detection unit 53 detectsthe positions and the orientations of the workpieces W, and theselection unit 59 selects the next target workpiece W to be taken out bythe robot 1. Then, the controller 4 implements control similar to thatdescribed above in order to take out the next target workpiece W.

When the operation for taking out the workpiece W by the robot 1 isperformed, the load collapse may occur. In other words, the positionsand the orientations of the workpieces W that have not been taken out bythe robot 1 may change. When the load collapse occurs, the secondthree-dimensional information includes many measurement points largelydeviated from the measurement points in the first three-dimensionalinformation. Thus, in the first three-dimensional information, thenumber of the specified measurement points arranged within thepredetermined distance range from the measurement points in the secondthree-dimensional information is reduced. Thus, the number of thespecified measurement points included in the first three-dimensionalinformation is reduced.

The determination unit 55 detects the number of measurement points otherthan the specified measurement point among the measurement points in thefirst three-dimensional information, in order to determine whether ornot the load collapse has occurred. The determination unit 55 determineswhether or not the number of measurement points other than the specifiedmeasurement points is larger than a predetermined determination valuerelated to the load collapse. The determination unit 55 can determinethat the load collapse has occurred when the number of measurementpoints other than the specified measurement points is larger than thedetermination value related to the load collapse. A value larger thanthe upper limit of the number of measurement points set to a singleworkpiece may be employed as the determination value related to the loadcollapse, for example.

Alternatively, the determination unit 55 may calculate the number ofmeasurement points in the second three-dimensional information, whichare arranged outside the predetermined distance range of the measurementpoints in the first three-dimensional information. When this number ofmeasurement points is larger than the determination value related to theload collapse, the determination unit 55 may determine that the numberof measurement points other than the specified measurement points amongthe measurement points in the first three-dimensional information islarger than the determination value related to the load collapse.

When it is determined that the number of measurement points other thanthe specified measurement points among the measurement points in thefirst three-dimensional information is larger than the determinationvalue related to the load collapse, the determination unit 55 transmitsa command for returning the workpiece W gripped by the hand 2 to thevicinity of the original position, to the operation control unit 43. Theoperation control unit 43 controls the robot 1 so as to return theworkpiece W gripped by the hand 2 to the vicinity of the originalposition before being gripped by the hand 2.

After the robot 1 returns the workpiece W to the vicinity of theoriginal position, the controller 4 implements the control for takingout the workpiece W from the operation for capturing the image of theworkpiece W. In other words, the range sensor 6 captures an image of theworkpieces W arranged in the container 9, the detection unit 53 detectsthe positions and the orientations of the workpieces W, and theselection unit 59 selects the next target workpiece W to be taken out bythe robot 1. Then, the controller 4 implements control similar to thatdescribed above in order to take out the next target workpiece W.

The robot system 5 of the present embodiment determines a state wherethe workpiece W is being gripped, when the robot 1 implements anoperation of lifting the workpiece W. The robot 1 can be controlled inaccordance with the state of the workpiece W being gripped. Thus, therobot system 5 can achieve a shorter cycle time for conveying theworkpiece W. For example, the robot 1 can be prevented from implementingthe operation of conveying the workpiece W to the target conveyancedestination position, in a state where the workpiece W is not actuallygripped. Alternatively, when the hand 2 grips the workpiece W other thanthe target workpiece or grips the target workpiece W and anotherworkpiece W, the workpiece(s) W can be prevented from being conveyed tothe target conveyance destination position. For example, the robot 1 canbe prevented from implementing an operation of returning theworkpiece(s) to the original position after the workpiece other than thetarget workpiece or two or more workpieces is (are) conveyed to thetarget conveyance destination position.

FIG. 11 shows a first side view of a part of the hand in a robot systemaccording to a first comparative example. In the first comparativeexample, a photoelectric sensor 85 is provided in order to determinewhether or not the hand 2 grips the workpiece WC. The photoelectricsensor 85 is fixed to the hand 2. The photoelectric sensor 85 emitslight as indicated by an arrow 95. It is determined that the hand 2grips the workpiece WC when the photoelectric sensor 85 detects lightreflected from the surface of the workpiece WC. It is determined thatthe hand 2 does not grip the workpiece WC when the photoelectric sensor85 does not detect the reflected light.

FIG. 12 is a second side view of a part of the hand in the robot systemof the first comparative example. The hand 2 of the comparative exampleis an electromagnet hand. Thus, the workpiece WC can be in anyorientation to be gripped. As illustrated in FIG. 12, the photoelectricsensor 85 cannot receive the reflected light in a case where theworkpiece WC is gripped while being inclined. Thus, the workpiece WC maybe determined to be not gripped, despite the fact that the workpiece WCis actually gripped. The first comparative example has a problem thatthe orientation of the photoelectric sensor 85 is difficult to set sincethe photoelectric sensor 85 is fixed to the hand 2. Alternatively, thephotoelectric sensor 85 may not be arranged in the orientation in whichthe workpieces WC gripped in various orientations can be detected.

FIG. 13 is a first side view of a part of the hand in a robot system ofa second comparative example. In the second comparative example, aphotoelectric sensor 86 is provided in order to determine whether or notthe hand 2 grips the workpiece WC. The photoelectric sensor 86 includesa light emitting element 86 a and a light receiving element 86 b. Thelight emitting element 86 a and the light receiving element 86 b can befixed to a support member and the like for example. The position and theorientation of the robot 1 can be controlled so as to place theworkpiece WC between the light emitting element 86 a and the lightreceiving element 86 b. The light emitting element 86 a emits light asindicated by the arrow 96. It can be determined that the hand 2 gripsthe workpiece WC when the light receiving element 86 b does not receivethe light.

FIG. 14 is a second side view of a part of the hand in the robot systemaccording to the second comparative example. As illustrated in FIG. 14,the light emitted from the light emitting element 86 a may reach thelight receiving element 86 b when the workpiece WC is gripped by thehand 2 in an orientation different from the desired orientation. Thus,the workpiece WC may be determined to be not gripped, despite the factthat the workpiece WC is actually gripped.

FIG. 15 is a third side view of a part of the hand in the robot systemof the second comparative example. As illustrated in FIG. 15, the hand 2may grip two or more workpieces WC. In such a case, the light emittedfrom the light emitting element 86 a can be blocked by the workpiece WC.However, with this control, a state where the plurality of workpieces WCare gripped cannot be detected. Referring to FIG. 11 and FIG. 12, thestate where the plurality of workpieces WC are gripped cannot bedetected also in the first comparative example.

Thus, the method of determining the state where a workpiece is grippedby using the photoelectric sensor may fail to accurately determine thestate where the hand grips the workpiece. On the other hand, the robotsystem 5 according to the present embodiment can accurately determinethe state in which the workpiece W is gripped by the hand 2. Theworkpiece W can be detected regardless of the orientation of theworkpiece W gripped by the hand 2. Alternatively, the state where two ormore workpieces W are gripped can be detected. Thus, the robot 1 can beprevented from being excessively operated.

In the second comparative example, the workpiece can be detected bymoving the hand to a plurality of positions and the orientations inconsideration of the plurality of states where the workpiece is gripped.However, this control disadvantageously leads to a long cycle time. Therobot system 5 of the present embodiment can detect the state where theworkpiece is gripped through the comparison between the firstthree-dimensional information and the second three-dimensionalinformation, and thus can detect the state where the workpiece isgripped in a short period of time.

Furthermore, the robot system 5 of the present embodiment requires nodevice for detecting the state where the workpiece is gripped, such as aphotoelectric sensor. The robot system 5 can detect the state where theworkpiece is gripped by using an output from the three-dimensionalsensor for detecting the position and the orientation of the workpiece.As described above, the robot system 5 of the present embodiment canaccurately confirm the state where the workpiece W is gripped with asimple configuration.

FIG. 16 is a flowchart illustrating a method of controlling the robotsystem of the present embodiment. The control illustrated in FIG. 16 canbe repeatedly implemented.

Referring to FIG. 2 and FIG. 16, in step 100, the operation control unit43 implements a first image capturing step for capturing an image of theworkpieces W with the range sensor 6. The generation unit 58 implementsa first generation step including setting measurement points on theworkpieces W based on an output from the range sensor 6, and generatingthe first three-dimensional information including the coordinate valuesof the measurement points. The generation unit 58 generates the firstthree-dimensional information based on the output from the range sensor6 that captured the image of the workpieces W before the robot 1implements the operation of gripping the target workpiece W.

In step 101, the detection unit 53 implements a first detection step ofdetecting the positions and the orientations of the workpieces W, byimplementing the model matching of comparing the first three-dimensionalinformation of the workpieces with the predetermined three-dimensionalshape data 46 of the workpiece W.

In step 102, the selection unit 59 implements a selection step ofselecting a target workpiece to be taken out by the robot 1, based onthe positions and the orientations of the workpieces W detected by thedetection unit 53.

In step 103, the operation control unit 43 implements an operationcontrol step of changing the position and the orientation of the robot1. In this operation control step, the robot 1 implements an operationof gripping and lifting the target workpiece W. Then, the robot 1 stopsin a state where the workpiece W is arranged in the vicinity of thecontainer 9.

In step 104, the range sensor 6 implements a second image capturing stepof capturing the image of the workpieces W. The generation unit 58implements a second generation step of generating the secondthree-dimensional information based on the output from the range sensor6 that captured the image of the workpieces W after the robot 1implements the operation of lifting the target workpiece W.

In step 105, the deletion unit 54 implements a deletion step of deletingthe information about at least a part of the measurement points includedin the first three-dimensional information. The deletion unit 54implements a step of comparing the positions of the measurement pointsin the first three-dimensional information with the positions of themeasurement points in the second three-dimensional information. Thedeletion unit 54 determines whether or not the measurement point in thesecond three-dimensional information is present within the predetermineddistance from each of the measurement points in the firstthree-dimensional information. The deletion unit 54 implements a step ofdetecting, as the specified measurement point, the measurement point inthe first three-dimensional information from which the measurement pointin the second three-dimensional information is present within thepredetermined distance range. The deletion unit 54 implements a step ofgenerating the third three-dimensional information in which theinformation about the coordinate value and the like of the specifiedmeasurement point is deleted from the first three-dimensionalinformation.

In step 106, the detection unit 53 implements a second detection step ofdetecting the position and the orientation of the workpiece included inthe third three-dimensional information after the processing in thedeletion step. The detection unit 53 implements model matching in whichthe third three-dimensional information about the workpiece is comparedwith the predetermined three-dimensional shape data 46 of the workpiece.The detection unit 53 detects the positions and the orientations of allthe workpieces W in the third three-dimensional information.

In step 108, the determination unit 55 implements a determination stepof determining the state after the operation of lifting the workpiece Wby the robot. In the determination step, it is determined whether or notthe workpiece W detected in the second detection step matches the targetworkpiece W. Here, the determination unit 55 determines whether or notonly the target workpiece W is taken out. When the workpiece W taken outdoes not match the target workpiece W in step 108, the control proceedsto step 121. For example, the control proceeds to step 121 when theworkpiece W lifted by the robot 1 is different from the target workpieceW, or when two or more workpieces W are lifted.

When it is determined that only the target workpiece W is taken out instep 108, the control proceeds to step 109. In step 109, an operationcontrol step is implemented. In this operation control step, theoperation control unit 43 controls the robot 1 so as to convey theworkpiece W gripped by the hand 2 to the predetermined conveyancedestination position.

Next, in step 110, the determination unit 55 determines whether or not apredetermined number of workpieces are taken out. In other words, it isdetermined whether or not the quantity of workpieces taken out reaches adetermination value for such a quantity. This determination value forthe quantity can be set in advance by the operator. For example, theoperator can set a small quantity such as five, as the determinationvalue. Alternatively, the operator may set the quantity corresponding toall the workpieces W accommodated in the container 9, as thedetermination value.

When the predetermined quantity of workpieces W are taken out in step110, the control is terminated. When the predetermined quantity ofworkpieces W are not taken out in step 110, the control proceeds to step111. In step 111, the generation unit 58 sets the current secondthree-dimensional information to be the first three-dimensionalinformation. In other words, the three-dimensional information as aresult of taking out the workpiece W is set to be the firstthree-dimensional information. In this control, only the targetworkpiece is detected in step 108. It can be determined that the loadcollapse does not occur in the workpieces W remaining in the container9. Thus, the second three-dimensional information generated after thetarget workpiece W is taken out can be used as the firstthree-dimensional information.

Then, the control proceeds to step 101 and the control for taking outthe workpiece W is implemented. In step 101, the detection unit 53detects the positions and the orientations of the workpieces W based onthe first three-dimensional information. In step 102, the selection unit59 selects the next target workpiece W. In this control, step 100 isomitted. The step of acquiring the three-dimensional information bycapturing an image of the workpieces with a three-dimensional camera canbe omitted. Thus, the cycle time can be shortened.

When the workpiece W taken out by the robot 1 does not match the targetworkpiece W in step 108, the control proceeds to step 121. In step 121,the determination unit 55 implements a determination step of determiningwhether or not a single workpiece is detected. The determination unit 55determines whether or not the hand 2 grips a single workpiece differentfrom the target workpiece. The determination unit 55 determines whetheror not the third three-dimensional information includes two workpieces.In other words, it is determined whether or not the hand 2 grips twoworkpieces. When one or two workpieces are detected in step 121, thecontrol proceeds to step 124.

In step 121, it may be determined whether or not two or more workpiecesare detected instead of determining whether or not two workpieces aredetected.

In step 124, the determination unit 55 sends a command for retuning theworkpiece W gripped by the hand 2 to the vicinity of the originalposition at which the workpiece W was placed before being gripped by thehand 2, to the operation control unit 43. The operation control unit 43controls the robot 1 so as to return the workpiece W to the vicinity ofthe original position. Then, the control proceeds to step 100. After therobot 1 returns the workpiece W to the vicinity of the originalposition, the control for taking out a workpiece is implemented from thefirst image capturing step. In step 100, the range sensor 6 implementsan image capturing step of capturing an image of a workpieces W. In step102, the detection step of detecting the positions and the orientationsof the workpieces is implemented. In step 103, the selection step ofselecting the next target workpiece W to be taken out by the robot 1 isimplemented. Then, the control similar to that described above isimplemented.

When one or two workpieces are not detected in step 121, the controlproceeds to step 122. In step 122, the determination unit 55 implementsthe determination step of determining whether or not the load collapsehas occurred. In this determination step, it is determined whether ornot the number of the measurement points in the first three-dimensionalinformation other than the specified measurement point is larger thanthe determination value related to the load collapse. The determinationunit 55 determines that the load collapse has occurred when the numberof the measurement points other than the specified measurement point islarger than the determination value related to the load collapse. Inthis case, the control proceeds to step 124.

In step 122, the determination unit 55 can determine that the loadcollapse does not occur when the number of the measurement points otherthan the specified measurement point is equal to or smaller than thedetermination value related to the load collapse. In this case, thecontrol proceeds to step 123.

In step 123, the determination unit 55 determines whether or not thehand 2 fails to grip the workpiece W. The determination unit 55implements a determination step of determining whether or not the numberof the measurement points in the first three-dimensional informationother than the specified measurement point is smaller than adetermination value related to failure in the gripping. Thedetermination unit 55 determines that the hand 2 fails to grip theworkpiece W, in a case where the number of the measurement points otherthan the specified measurement point is smaller than the determinationvalue related to failure in the gripping. In this case, the controlreturns to step 100. The control is implemented from the first imagecapturing step of capturing the image of the workpieces W. When it isdetermined that the hand 2 successfully grips the workpiece W in step123, the control proceeds to step 124.

In the control illustrated in FIG. 16, the control for taking out theworkpiece is implemented until the predetermined number of workpiecesare taken out as illustrated in step 110, but the embodiment is notlimited to this. The control may be terminated when all the workpieces Warranged in the container 9 are taken out. For example, the control maybe terminated when no workpiece arranged in the case is detected in step101.

The three-dimensional sensor of the present embodiment includes the twotwo-dimensional cameras, but the embodiment is not limited to this. Thethree-dimensional sensor may include three or more two-dimensionalcameras. By adopting this configuration where the three-dimensionalsensor includes three or more cameras, even when part of the images isunclear due to halation and the like, the three-dimensional informationcan be generated based on images captured by other cameras. Thethree-dimensional sensor of the present embodiment includes theprojector, but the projector may not be provided. Furthermore, thethree-dimensional sensor may include any appropriate sensor that canacquire information about a distance to a workpiece. For example, thethree-dimensional sensor may be a Time of Flight (TOF) camera thatcaptures a distance image by using the time of flight method.

The range sensor 6 of the present embodiment is fixed to the supportmember 83, but the embodiment is not limited to this, and can bearranged so as to be capable of capturing an image of workpieces. Forexample, the range sensor may be fixed to the wrist of the robot so asto integrally move with the wrist.

The hand 2 of the present embodiment is a magnetic hand, but theembodiment is not limited to this. Any operation tool for gripping andreleasing a workpiece can be employed as the hand. For example, a handthat has a plurality of craws, and grips a workpiece with the crawsclamping the workpiece, a hand that grips a workpiece with a suction padwith air in the suction pad sucked, or the like can be employed.

According to the robot system and the control method of the robot systemof the aspect of the present disclosure, the state where the hand gripsa workpiece can be accurately determined when workpieces loaded in bulkare taken out.

In each control described above, the sequence of steps may be changed asappropriate, within such a range that the functionality and operationare not changed.

The above-described embodiments can be combined as appropriate.Identical or equivalent parts are given identical reference numerals inthe above-described drawings. Note that the above-described embodimentsare merely examples and are not intended to limit the invention. Changesto the embodiments as indicated in the claims are also included in theembodiments.

The invention claimed is:
 1. A robot system that takes out each ofworkpieces loaded in bulk inside a case, the robot system comprising: ahand that grips a workpiece, a robot that moves the hand, athree-dimensional sensor that acquires information about distances tothe workpieces, and a controller that controls the hand and the robot,wherein the controller includes: an operation control unit that sendsoperation commands to the hand and the robot; a generation unit thatgenerates three-dimensional information including information aboutpositions of measurement points set to the workpieces based on an outputfrom the three-dimensional sensor; a storage unit that storesthree-dimensional shape data of the workpieces; a detection unit thatdetects positions and orientations of the workpieces by implementingmodel matching in which the three-dimensional information about theworkpieces is compared with the three-dimensional shape data; aselection unit that selects a target workpiece to be taken out by therobot based on the positions and the orientations of the workpieces; adeletion unit that deletes information about at least part of themeasurement points included in the three-dimensional information; and adetermination unit that determines a state after the robot implements anoperation of lifting the workpiece, the generation unit generates firstthree-dimensional information based on an output from thethree-dimensional sensor that has captured an image of the workpiecesbefore the robot implements an operation of gripping the targetworkpiece, the operation control unit controls the robot so as to stopafter implementing an operation of gripping and lifting the targetworkpiece selected based on the first three-dimensional information, thegeneration unit generates second three-dimensional information based onan output from the three-dimensional sensor that has captured an imageof the workpieces after the robot implements the operation of liftingthe target workpiece, the deletion unit compares a position of themeasurement point in the first three-dimensional information with aposition of the measurement point in the second three-dimensionalinformation, detects, as a specified measurement point, the measurementpoint in the first three-dimensional information from which themeasurement point in the second three-dimensional information is presentwithin a predetermined distance range, and generates thirdthree-dimensional information in which information about the specifiedmeasurement point is deleted from the first three-dimensionalinformation, the detection unit detects a position and an orientation ofa workpiece included in the third three-dimensional information, thedetermination unit determines whether or not the workpiece included inthe third three-dimensional information matches the target workpiece,and when the workpiece included in the third three-dimensionalinformation matches the target workpiece, the operation control unitcontrols the robot so as to convey the workpiece gripped by the hand toa predetermined conveyance destination position.
 2. The robot systemaccording to claim 1, wherein the operation control unit controls therobot so as to return the workpiece gripped by the hand to a vicinity ofan original position of the workpiece before being gripped by the hand,when the determination unit determines that the workpiece included inthe third three-dimensional information is different from the targetworkpiece or that two or more workpieces are included in the thirdthree-dimensional information, and after the robot returns the workpieceto the vicinity of the original position, the three-dimensional sensorcaptures an image of the workpieces arranged in the case, the detectionunit detects positions and orientations of the workpieces, and theselection unit selects a next target workpiece to be taken out by therobot.
 3. The robot system according to claim 1, wherein thedetermination unit determines whether or not number of measurementpoints in the first three-dimensional information other than specifiedmeasurement points is larger than a predetermined determination valuerelated to load collapse, when the determination unit determines thatthe number of measurement points other than specified measurement pointsis larger than the determination value related to the load collapse, theoperation control unit controls the robot so as to return the workpiecegripped by the hand to the vicinity of the original position of theworkpiece before being gripped by the hand, and after the robot returnsthe workpiece to the vicinity of the original position, thethree-dimensional sensor captures an image of the workpieces arranged inthe case, the detection unit detects positions and orientations of theworkpieces, and the selection unit selects a next target workpiece to betaken out by the robot.
 4. The robot system according to claim 1,wherein the determination unit determines whether or not number ofmeasurement points in the first three-dimensional information other thanspecified measurement points is smaller than a predetermineddetermination value related to failure in gripping, and when thedetermination unit determines that the number of measurement pointsother than specified measurement points is smaller than thedetermination value related to the failure in the gripping, thethree-dimensional sensor captures an image of the workpieces arranged inthe case, the detection unit detects positions and orientations of theworkpieces, and the selection unit selects a next target workpiece to betaken out by the robot.
 5. A control method of a robot system thatincludes a robot and a hand and takes out each of workpieces loaded inbulk in a case, the method comprising: an operation control step ofchanging a position and an orientation of the robot; an image capturingstep of capturing an image of the workpieces with a three-dimensionalsensor that acquires information about distances to the workpieces; ageneration step of generating three-dimensional information includinginformation about positions of measurement points set to the workpiecesbased on an output from the three-dimensional sensor; a detection stepof detecting the positions and the orientations of the workpieces byimplementing model matching in which the three-dimensional informationabout the workpieces is compared with three-dimensional shape data ofthe workpieces; a selection step of selecting a target workpiece to betaken out by the robot based on the positions and the orientations ofthe workpieces; a deletion step of deleting information about at leastpart of the measurement points included in the three-dimensionalinformation; and a determination step of determining a state after therobot implements an operation of lifting the workpiece, wherein thegeneration step includes generating first three-dimensional informationbased on an output from the three-dimensional sensor that has capturedan image of the workpiece before the robot implements an operation ofgripping the target workpiece, the operation control step includes astep of controlling the robot so as to stop after implementing anoperation of gripping and lifting the target workpiece selected based onthe first three-dimensional information, the generation step includesgenerating second three-dimensional information based on an output fromthe three-dimensional sensor that has captured an image of theworkpieces after the robot has implemented the operation of lifting thetarget workpiece, the deletion step includes a step of comparing aposition of the measurement point in the first three-dimensionalinformation with a position of the measurement point in the secondthree-dimensional information, a step of detecting, as a specifiedmeasurement point, the measurement point in the first three-dimensionalinformation from which the measurement point in the secondthree-dimensional information is present within a predetermined distancerange, and a step of generating third three-dimensional information inwhich information about the specified measurement point is deleted fromthe first three-dimensional information, the detection step includes astep of detecting a position and an orientation of a workpiece includedin the third three-dimensional information, the determination stepincludes a step of determining whether or not the workpiece included inthe third three-dimensional information matches the target workpiece,and the operation control step includes a step of controlling, when theworkpiece included in the third three-dimensional information matchesthe target workpiece, the robot so as to convey the workpiece gripped bythe hand to a predetermined conveyance destination position.
 6. Thecontrol method of a robot system according to claim 5, wherein the robotis controlled to return the workpiece gripped by the hand to a vicinityof an original position of the workpiece before being gripped by thehand, in response to determining in the determination step that theworkpiece included in the third three-dimensional information isdifferent from the target workpiece or that two or more workpieces areincluded in the third three-dimensional information, and after the robotreturns the workpiece to the vicinity of the original position, an imagecapturing step of capturing an image of the workpieces arranged in thecase with the three-dimensional sensor is performed, a detection step ofdetecting positions and orientations of the workpieces is performed, anda selection step of selecting a next target workpiece to be taken out bythe robot is performed.
 7. The control method of a robot systemaccording to claim 5, wherein the determination step includesdetermining whether or not number of measurement points in the firstthree-dimensional information other than specified measurement points islarger than a predetermined determination value related to loadcollapse, in response to determining that the number of measurementpoints other than specified measurement points is larger than thedetermination value related to the load collapse, the robot iscontrolled so as to return the workpiece gripped by the hand to thevicinity of the original position of the workpiece before being grippedby the hand, and after the robot returns the workpiece to the vicinityof the original position, an image capturing step of capturing an imageof the workpieces arranged in the case with the three-dimensional sensoris performed, a detection step of detecting positions and orientationsof the workpieces is performed, and a selection step of selecting a nexttarget workpiece to be taken out by the robot is performed.
 8. Thecontrol method of a robot system according to claim 5, wherein thedetermination step includes determining whether or not number ofmeasurement points in the first three-dimensional information other thanspecified measurement points is smaller than a predetermineddetermination value related to failure in gripping, and in response todetermining that the number of measurement points other than specifiedmeasurement points is smaller than the determination value related tothe failure in the gripping, an image capturing step of capturing animage of the workpieces arranged in the case with the three-dimensionalsensor is performed, a detection step of detecting positions andorientations of the workpieces is performed, and a selection step ofselecting a next target workpiece to be taken out by the robot isperformed.